A Online Appendix to: Analysis and Optimization for Boolean Expression Indexing

String tokenization using q-grams maps the string into a high-dimensional vector space model, in which the domain of each dimension is binary. The size of this space is exponential in the length of q-grams. For instance, q-grams of size three results in a space with 26 dimensions. The vector space model representation of a tokenized string (e.g., {‘str’, ‘tri’, ‘rin’, ‘ing’}) can be expressed by setting dimensions associated to each of its grams (e.g., ‘str’, ‘tri’, ‘rin’, and ‘ing’) to 1 and everything else to 0. Alternatively, we can concisely express this as binary equality predicates, e.g., the predicate with dimension (i.e., attribute) ‘str’ must be equal to 1 and ignoring dimension with value 0. Therefore, instead of expressing a tokenized string as a vector of 0s or 1s, we can express it as a set of equality predicates. We can also take this one step further by going beyond binary domains to capture more interesting relationships among q-grams and as a byproduct reducing the space dimensionality. The original model mapped each q-gram to a dimension, instead we can map only the prefix of each q-gram to a dimension and map the rest as a value in the corresponding dimension. For example, the q-gram ‘str’ is mapped to the quality predicate [‘st’ = ‘r’], ‘st’ now represents the predicate’s dimension and ‘r’ is mapped to the value in this dimension. In this new mapping, the number of dimension is reduced from 26 to 26, and there is a new opportunity to express similarity among overlapping q-grams. For example, the q-grams ‘str’ and ‘ste’ are now both mapped to dimensions represented by ‘st’ (already signifying a similarity among these two q-grams), but also the value ‘r’ and ‘e’ can play an important role. For instance, since the letters ‘r’ and ‘e’ are adjacent on standard U.S. keyboard, then it is possible, that due to typing error q-grams ‘str’ was entered as ‘ste’. This relationship can be captured by mapping ‘r’ to ‘e’ to spatially close values.